FEAT: Chain-of-Thought Rendering for Attack Results by ValbuenaVC · Pull Request #2269 · microsoft/PyRIT

ValbuenaVC · 2026-07-23T22:09:18Z

Description

Adds opt-in rendering of OpenAI Responses API reasoning summaries across PyRIT’s output layer.

This PR:

Adds include_reasoning_trace=False to conversation and attack-result APIs.
Renders provider-generated reasoning summaries while keeping them hidden by default.
Validates persisted reasoning pieces against the OpenAI Responses reasoning-item shape.
Prevents reasoning content from leaking after conversion to another data type.
Renders reasoning summaries in subdued gray for Pretty output.
Renders escaped reasoning headers in Markdown.
Supports objective, pruned, and adversarial attack conversations.
Clearly labels summaries as provider-generated summaries, not raw hidden chain-of-thought.

No breaking behavior is introduced because reasoning remains opt-in.

Background and Scope

The original story also requested investigation into JSON Schema adoption for scorers and attacks.

That infrastructure and adoption already landed through:

[FEAT] Support JSON Schema in Responses #1177 — OpenAI Chat/Responses JSON Schema support
FEAT: Adding Json Schema Pipeline to SeedPrompts #1432 — SeedPrompt schema pipeline, persistence, capability adaptation, and refusal-scorer adoption
FEAT: Adding json schema to all scorer datasets #2017 — schema adoption across additional scorer datasets
MAINT: Unify adversarial-chat JSON schema across Crescendo, TAP, and PAIR #2039 and MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback #2053 — shared adversarial-chat schema across Crescendo, TAP/PAIR, Red Teaming, and related attacks

Schemas remain domain-specific; this PR does not introduce a universal response schema or change target/scorer/attack semantics.

CoPyRIT already maps persisted reasoning pieces into its existing Reasoning panel and has mapper/component test coverage. This PR adds the missing PyRIT output parity. CoPyRIT currently has no scenario-results UI, so scenario frontend rendering is outside this PR.

Tests and Documentation

Added or expanded unit coverage for:

Pretty and Markdown conversation rendering
Attack objective, pruned, and adversarial conversations
Empty and malformed reasoning summaries
Exact OpenAI Responses payload validation
Original/converted data-type leakage prevention
Exact header literals, spacing, and Pretty colors
Public helper argument forwarding

Local results:

171 passed
pyrit.output: 98% statement coverage
Ruff and formatting checks pass
All required PR checks currently pass, including frontend unit, E2E, lint, and type-check workflows

Updated the paired output documentation with conversation, attack, and scenario examples and clarification that OpenAI exposes summaries rather than raw chain-of-thought.

…lts.

Copilot

Pull request overview

Adds opt-in provider reasoning-summary rendering across conversation, attack-result, and scenario-result output.

Changes:

Parses and validates OpenAI reasoning-summary payloads.
Adds pretty and Markdown rendering with reasoning hidden by default.
Propagates reasoning options through helpers and adds unit coverage.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`pyrit/output/conversation/base.py`	Adds reasoning filtering and parsing.
`pyrit/output/conversation/pretty.py`	Renders pretty reasoning blocks.
`pyrit/output/conversation/markdown.py`	Renders Markdown reasoning blocks.
`pyrit/output/attack_result/base.py`	Extends the rendering contract.
`pyrit/output/attack_result/pretty.py`	Propagates reasoning through pretty output.
`pyrit/output/attack_result/markdown.py`	Propagates reasoning through Markdown output.
`pyrit/output/scenario_result/base.py`	Extends the scenario rendering contract.
`pyrit/output/scenario_result/pretty.py`	Adds scenario-level reasoning output.
`pyrit/output/helpers.py`	Exposes reasoning options in helpers.
`tests/unit/output/conftest.py`	Adds shared reasoning fixtures.
`tests/unit/output/conversation/test_reasoning.py`	Tests parsing, visibility, and formatting.
`tests/unit/output/attack_result/test_pretty.py`	Tests pretty attack reasoning.
`tests/unit/output/attack_result/test_markdown.py`	Tests Markdown attack reasoning.
`tests/unit/output/scenario_result/test_pretty.py`	Tests scenario reasoning output.
`tests/unit/output/test_helpers.py`	Tests helper argument forwarding.

…c to _render_attack_reasoning_summaries_async for clarity.

…://github.com/ValbuenaVC/PyRIT into vvalbuena-microsoft-plan-cot-output-rendering Merge latest changes from main.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pyrit/output/scenario_result/pretty.py:341

When scenario reasoning is enabled, this nested attack printer renders each full objective conversation, and PrettyConversationMemoryPrinter displays every image_path piece in notebooks. Because this printer is always created with its default blur_images=False and output_scenario_async exposes no blur option, requesting reasoning summaries can unexpectedly display unblurred attack images with no way for callers to opt into the safety control available on output_attack_async. Please either render only the reasoning blocks or plumb the image-blur settings through the scenario API and nested printer.

        attack_result_printer = PrettyAttackResultMemoryPrinter(
            sink=sink,
            width=width,
            indent_size=indent_size,
            enable_colors=enable_colors,
        )

behnam-o

Looks great! I just have these two comments, one about whether scenario results should even expose this feature and another about how we seem to silently just not print a reasoning section if the model hasn't "reasoned" (I think in that case, it's good to expose/render this fact rather than treating it the same as if the model does not "support/generate" reasoning at all)

behnam-o · 2026-07-27T16:45:23Z

-            data = json.loads(reasoning_value)
-        except (json.JSONDecodeError, TypeError):
+        summary = self._extract_reasoning_summary(reasoning_value)
+        if not summary:


Could we distinguish between an absent reasoning piece and a valid reasoning piece with an empty summary?

When include_reasoning_trace=True:

If no reasoning piece exists, rendering nothing makes sense because reasoning may be unsupported or may not have been requested.

If a valid reasoning piece exists but its summary list is empty, silently omitting it hides useful state and may leave an empty message section. Could we render something explicit, such as [No reasoning summary was returned by the provider.]?

If the payload is malformed, continuing to raise ValueError seems appropriate.

I'm not sure I understand your second point. What useful state would be hidden in that context? This is only called on line 114 (for piece in pieces, if this is a reasoning piece, then render the summary). Are you saying we might get a response that's annotated as reasoning (piece.original_value_data_type) and we want the user to be aware of this even when there's no contents in summary?

yeah exactly. This is how I imagine an end user would use this feature:

I execute an attack and get a response, I'm curious how the model came to that conclusion or what how my prompt was processed in the reasoning phase, so I try to print not only the response, but the reasoning behind it too.

Because this case (explicitly setting include_reasoning) is intentionally done by a user, I think it makes sense to show exactly the reasoning bits, even if it's empty, so they know the model/api itself did not return anything in the reason fields, not that printing/UI is not working okay ...

Makes perfect sense to compartmentalize concerns in that case. Will update

behnam-o · 2026-07-27T16:53:33Z

it might also be a good idea to include a sample prompt and output that triggers a reasoning piece and show how it gets rendered. for example, I used this snippet to see what the final output looks like, I think it's worth having a cell in the output notebook in the docs

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…://github.com/ValbuenaVC/PyRIT into vvalbuena-microsoft-plan-cot-output-rendering Resolve merge confict. Adds documentation update to remote.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

…verts files to main.

hannahwestra25 · 2026-07-27T21:46:53Z

+        # PyRIT adds these visible tags. They label a provider-generated summary
+        # and do not represent raw model chain-of-thought.
+        block_lines = [
+            r"\<reasoning-summary\>",


nit: is there a reason we're using this html type format ? it just doesn't really match the rest of the output imo. like why not just have a section labeled reasoning summary

Not really. Any objections to something like a bold "💭 Reasoning" heading?

This is what it looks like inline

And here's a markdown example

cool! i like that

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-07-27T21:54:51Z

@@ -5,7 +5,7 @@
 #       extension: .py
 #       format_name: percent


nit: add something to open ai responses notebook

PyRIT/doc/code/targets/2_openai_responses_target.ipynb

Line 4 in 3da58f8

"cell_type": "markdown",

hannahwestra25 · 2026-07-27T21:58:29Z

+        Raises:
+            ValueError: If the value is not valid JSON or does not match the expected
+                OpenAI Responses reasoning-summary shape.
+        """


Should we catch this valueerror in pretty.py::_render_reasoning_summary and markdown.py::_format_reasoning_summary ? Since include_reasoning_trace is opt-in and reasoning data is provider-generated (subject to schema drift across OpenAI API versions, or a future non-OpenAI target tagging pieces data_type="reasoning" with a different shape), one malformed reasoning piece will abort rendering of the entire conversation/attack/scenario report rather than just that piece

Really good catch, I agree

hannahwestra25 · 2026-07-27T22:00:15Z

plans to add this to the scanner ?

Co-authored-by: hannahwestra25 <hannahwestra@microsoft.com>

…://github.com/ValbuenaVC/PyRIT into vvalbuena-microsoft-plan-cot-output-rendering Merging in remote changes from PR review.

ValbuenaVC · 2026-07-27T22:14:17Z

plans to add this to the scanner ?

Not unless you feel it belongs there. Do users want to see the full reasoning trace during a scanner run? I'm not opposed to it, but I assumed the answer was no, though I may be wrong.

FEAT: Chain-of-Thought Rendering for attack results and scenario resu…

bf1d9c4

…lts.

ValbuenaVC requested a review from Copilot July 23, 2026 22:09

Merge branch 'main' into vvalbuena-microsoft-plan-cot-output-rendering

69a9d53

Copilot started reviewing on behalf of ValbuenaVC July 23, 2026 22:13 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread pyrit/output/scenario_result/pretty.py Outdated

Merge branch 'main' into vvalbuena-microsoft-plan-cot-output-rendering

eb3c47f

ValbuenaVC marked this pull request as ready for review July 24, 2026 22:40

Merge branch 'main' into vvalbuena-microsoft-plan-cot-output-rendering

67a1e1d

ValbuenaVC changed the title ~~[DRAFT] FEAT: Chain-of-Thought Rendering for Attack Results and Scenario Results~~ FEAT: Chain-of-Thought Rendering for Attack Results and Scenario Results Jul 24, 2026

behnam-o self-assigned this Jul 24, 2026

Victor Valbuena added 2 commits July 24, 2026 16:56

FEAT: Generated documentation and renamed _render_attack_results_asyn…

e31c427

…c to _render_attack_reasoning_summaries_async for clarity.

Merge branch 'vvalbuena-microsoft-plan-cot-output-rendering' of https…

389a551

…://github.com/ValbuenaVC/PyRIT into vvalbuena-microsoft-plan-cot-output-rendering Merge latest changes from main.

ValbuenaVC requested a review from Copilot July 24, 2026 23:58

Copilot started reviewing on behalf of ValbuenaVC July 24, 2026 23:59 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

behnam-o reviewed Jul 27, 2026

View reviewed changes

ValbuenaVC and others added 4 commits July 27, 2026 11:14

Merge branch 'main' into vvalbuena-microsoft-plan-cot-output-rendering

611323b

DOC: Add reasoning rendering example

dd893d6

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into vvalbuena-microsoft-plan-cot-output-rendering

210da02

Merge branch 'vvalbuena-microsoft-plan-cot-output-rendering' of https…

fba8128

…://github.com/ValbuenaVC/PyRIT into vvalbuena-microsoft-plan-cot-output-rendering Resolve merge confict. Adds documentation update to remote.

ValbuenaVC requested a review from Copilot July 27, 2026 21:32

Copilot started reviewing on behalf of ValbuenaVC July 27, 2026 21:32 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread pyrit/output/scenario_result/pretty.py Outdated

Comment thread doc/code/output/0_output.ipynb Outdated

Victor Valbuena added 2 commits July 27, 2026 14:40

FEAT: Removed include_reasoning_trace from scenario-result output. Re…

da24436

…verts files to main.

FIX: Reverted unit tests for scenario_result pretty printing

e8664af

hannahwestra25 reviewed Jul 27, 2026

View reviewed changes

Comment thread pyrit/output/conversation/base.py Outdated

hannahwestra25 reviewed Jul 27, 2026

View reviewed changes

Comment thread pyrit/output/conversation/base.py

FIX: Remove scenario reasoning helper forwarding

2074712

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

hannahwestra25 reviewed Jul 27, 2026

View reviewed changes

ValbuenaVC and others added 3 commits July 27, 2026 15:08

Update pyrit/output/conversation/base.py

76541a9

Co-authored-by: hannahwestra25 <hannahwestra@microsoft.com>

FEAT: Removed converted prompt type checking

b1a790e

Merge branch 'vvalbuena-microsoft-plan-cot-output-rendering' of https…

daf4d4a

…://github.com/ValbuenaVC/PyRIT into vvalbuena-microsoft-plan-cot-output-rendering Merging in remote changes from PR review.

FIX precommit and accidental file change

985a92b

ValbuenaVC changed the title ~~FEAT: Chain-of-Thought Rendering for Attack Results and Scenario Results~~ FEAT: Chain-of-Thought Rendering for Attack Results Jul 27, 2026

FEAT Changed reasoning format from tags to headers with emojis

21d437e

Uh oh!

Conversation

ValbuenaVC commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Background and Scope

Tests and Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

behnam-o left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

behnam-o commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hannahwestra25 commented Jul 27, 2026

Uh oh!

ValbuenaVC commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ValbuenaVC commented Jul 23, 2026 •

edited

Loading